Multimodal Interfaces - A Generic Design Approach

Integrating new input-output modalities, such as speech, gaze, gestures, haptics, etc., in user interfaces is currently considered as a significant potential contribution to implementing the concept of Universal Access (UA) in the Information Society; see (Oviatt, 2003), for instance. UA in this context means providing everybody, including handicapped users, with easy human-computer interaction in any context of use, especially mobility. However, the cost of developing an appropriate specific multimodal user interface for each interactive software is prohibitive. A generic design methodology and generic reusable components are needed to master the complexity of the design and development of interfaces that allow flexible use of alternative modalities and meaningful combinations of modalities, according to the constraints in the interaction environment or the user's motor and perceptual capabilities. We present a design approach meant to facilitate the development of generic multimodal user interfaces, and based on best practice in software and user interface design and architecture. At present, specific on the shelf components are available that can process and interpret data from a wide range of input devices reliably. However, these components are monomodal in the sense that they are dedicated to a specific medium and modality. There is not yet, outside research laboratory prototypes, any software platform capable of interpreting multimodal input data. Symmetrically, software on the market is available for the generation of monomodal output messages conveyed through various media, whereas the generation of multimodal presentations is not yet supported. In the next section, we present a design approach that makes it possible to: − Interpret users' multimodal commands or manipulations/actions using partial monomodal interpretations, each interpretation being elaborated by a dedicated component which processes the specific input data stream transmitted through one of the available media. This treatment may be viewed as a fusion process of events or data. − Match these global interpretations with appropriate functions in the current application software, or translate them into appropriate commands (i.e., execution calls of the appropriate functions in the kernel of the considered software). − As regards system multimodal outputs, break up the information content of system messages into chunks, assign the resulting data chunks to appropriate modalities, and input each of them into the relevant monomodal generation/presentation component. This treatment is often viewed as a data fission process in contrast with multimodal input fusion. In most current applications, stereotyped system messages only need to be implemented in order to achieve efficient …